User interface for displaying object-based indications in an autonomous driving system

ABSTRACT

A vehicle has a plurality of control apparatuses, a user input, a geographic position component, an object detection apparatus, memory, and a display. A processor is also included and is programmed to receive the destination information, identify a route, and determine the current geographic location of the vehicle. The processor is also programmed to identify an object and object type based on object information received from the object detection apparatus and to determine at least one warning characteristic of the identified object based on at least one of: the object type, a detected proximity of the detected object to the vehicle, the location of the detected object relative to predetermined peripheral areas of the vehicle, the current geographic location of the vehicle, and the route. The processor is also configured to select and display on the display an object warning image based on the at least one warning characteristic.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/295,433, filed Oct. 17, 2016, which is a continuation ofU.S. patent application Ser. No. 14/542,799, filed Nov. 17, 2014, nowissued as U.S. Pat. No. 9,501,058, which is a continuation of U.S.patent application Ser. No. 14/171,904, filed Feb. 4, 2014, now issuedas U.S. Pat. No. 8,903,592, which is a continuation of U.S. patentapplication Ser. No. 13/796,037, filed Mar. 12, 2013, now issued as U.S.Pat. No. 8,676,431, the entire disclosures of which are incorporatedherein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The invention relates generally to user interface applications fornavigation and autonomous driving systems. More specifically, userinterfaces for displaying information related to vehicles within thevicinity of an automobile.

Description of Related Art

Autonomous vehicles use various computing systems to transportpassengers from one location to another. Some autonomous vehicles mayrequire some initial input from an operator, such as a pilot, driver, orpassenger while other systems may require continuous input. Othersystems, for example autopilot systems, may be used only when the systemhas been engaged, thus the operator may switch from a manual to anautonomous mode where the vehicle drives itself.

A key component of an autonomous vehicle is the perception system, whichallows the vehicle to perceive and interpret its surroundings during atrip. When the autonomous system is engaged, the system will makevarious decisions during the trip, for example, speed up, slow down,stop, etc. The operator may be unaware of the calculations or“reasoning” behind why the autonomous vehicle is taking some particularaction. In order to feel safe and confident, the operator may want toknow what the vehicle is planning to do in the immediate future and tobe informed as to at least some of the factors influencing the system'sreasoning.

Navigation systems may include electronic displays which appear to zoomin or out according to a vehicle's speed of travel to enable to user toidentify where the vehicle may be within the next few seconds. Some ofthese systems provide real-time traffic information received via radioor satellite signals. However, these systems do not provide for thedisplay of the speed, actual location of other vehicles or obstacles, orother useful information related to such vehicles or obstacles.

BRIEF SUMMARY OF THE INVENTION

An aspect of the present disclosure relates to a vehicle having aplurality of control apparatuses including a braking apparatus, anacceleration apparatus, and a steering apparatus. The vehicle furtherhas a user input device for inputting destination information, ageographic position component for determining the current location ofthe vehicle, an object detection apparatus for detecting and identifyinga type of an object in or proximate to a roadway, memory for storing adetailed roadway map including roadways, traffic signals, andintersections, and an electronic display for displaying information to apassenger. A processor is also included in the vehicle and is programmedto receive the destination information, identify a route to thedestination, and determine, from location information received from thegeographic position component and the stored map information, thecurrent geographic location of the vehicle. The processor is alsoprogrammed to identify an object and object type based on objectinformation received from the object detection apparatus and todetermine at least one warning characteristic of the identified objectbased on at least one of: the object type, a detected proximity of thedetected object to the vehicle, the location of the detected objectrelative to predetermined peripheral areas of the vehicle, the currentgeographic location of the vehicle, and the route. The processor is alsoconfigured to select an object warning image to be displayed based onthe at least one warning characteristic and display the selected objectwarning image on the electronic display.

The processor can be further configured to identify a change in positionof the identified object over time. In such an example, the at least onewarning characteristic can relate to deceleration of the identifiedobject when the identified object is positioned in front of the subjectvehicle. In another example, the at least one warning characteristic canrelate to one of the presence of the identified object toward the rearof the vehicle and within a predetermined distance thereof and thepresence of the object within a blind spot of the vehicle.

Another aspect of the present disclosure relates to a vehicle having anobject detection apparatus for detecting and identifying a type of anobject in or proximate to a roadway and a location of the object and anelectronic display for displaying information to a passenger. Thevehicle also includes a processor programmed to identify an object andobject type based on object information received from the objectdetection apparatus and determine a relative position of the object tothe vehicle. The processor is further programmed to determine at leastone warning characteristic of the identified object based on at leastone of: the object type, a detected proximity of the detected object tothe vehicle, and the relative position of the detected object. Theprocessor is further programmed to determine one of a plurality ofpredetermined peripheral areas of the vehicle to associate with thewarning characteristic based on the relative position of the detectedobject and to select an object warning image and a warning locationimage to be displayed based on the at least one warning characteristicand the predetermined peripheral area associated with the warningcharacteristic. The processor is further configured to display theselected object warning image and the selected warning location image onthe electronic display and a preselected safe indication imageassociated with the predetermined peripheral areas not associated withthe warning characteristic.

Another aspect of the present disclosure relates to a method forselecting images for display on a display apparatus of a vehicle. Themethod includes receiving destination information from a user inputdevice, identifying a route to the destination, receiving locationinformation from a geographic position component, accessing stored mapinformation including roadways, traffic signals, and intersections, anddetermining, from the location information and the stored mapinformation, the current geographic location of the vehicle. The methodalso includes identifying an object of a roadway and an object typebased on object information received from an object detection apparatus,and determining at least one warning characteristic of the identifiedobject based on at least one of: the object type, a detected proximityof the detected object to the vehicle, the location of the detectedobject relative to predetermined peripheral areas of the vehicle, thecurrent geographic location of the vehicle, and the route. The methodalso includes selecting an object warning image to be displayed based onthe at least one warning characteristic and displaying the selectedobject warning image on the electronic display.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of a system in accordance with an aspectof the disclosure.

FIG. 2 is an exemplary design of the interior of an autonomous vehiclein accordance with an aspect of the disclosure.

FIG. 3 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 4 is another exemplary screen shot in accordance with an aspect ofthe disclosure.

FIG. 5 is another exemplary screen shot in accordance with an aspect ofthe disclosure.

FIG. 6 is an illustration of a notification image that can be used inconnection with an aspect of the disclosure.

FIG. 7 is an illustration of a variation of the notification image ofFIG. 6.

FIG. 8 is another configuration of the notification image of FIG. 7.

FIG. 9 is another configuration of the notification image of FIG. 7.

FIG. 10 is an illustration of an alternative notification image that canbe used in connection with an aspect of the disclosure.

FIG. 11 is an illustration of an alternative notification image that canbe used in connection with an aspect of the disclosure.

FIG. 12 is an illustration of an alternative notification image that canbe used in connection with an aspect of the disclosure.

FIG. 13 is an illustration of an alternative notification image that canbe used in connection with an aspect of the disclosure.

FIG. 14 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 15 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 16 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 17 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 18 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 19 is an exemplary screen shot in accordance with an aspect of thedisclosure.

FIG. 20 is an exemplary screen shot in accordance with an aspect of thedisclosure.

DETAILED DESCRIPTION

Aspects, features and advantages of the invention will be appreciatedwhen considered with reference to the following description of exemplaryembodiments and accompanying figures. The same reference numbers indifferent drawings may identify the same or similar elements.Furthermore, the following description is not limiting; the scope of theinvention is defined by the appended claims and equivalents.

As shown in FIG. 1, an autonomous driving system 100 in accordance withone aspect of the invention includes a vehicle 101 with variouscomponents. The vehicle may have one or more computers, such as computer110 containing a processor 120, memory 122 and other componentstypically present in general purpose computers.

The memory 122 stores information accessible by processor 120, includinginstructions 124 and data 126 that may be executed or otherwise used bythe processor 120. The memory 122 may be of any type capable of storinginformation accessible by the processor, including a computer-readablemedium, or other medium that stores data that may be read with the aidof an electronic device, such as a hard-drive, memory card, ROM, RAM,DVD or other optical disks, as well as other write-capable and read-onlymemories. Systems and methods may include different combinations of theforegoing, whereby different portions of the instructions and data arestored on different types of media.

The instructions 124 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computer codeon the computer-readable medium. In that regard, the terms“instructions” and “programs” may be used interchangeably herein. Theinstructions may be stored in object code format for direct processingby the processor, or in any other computer language including scripts orcollections of independent source code modules that are interpreted ondemand or compiled in advance. Functions, methods and routines of theinstructions are explained in more detail below.

The data 126 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 124. For instance, although the systemand method is not limited by any particular data structure, the data maybe stored in computer registers, in a relational database as a tablehaving a plurality of different fields and records, XML documents orflat files. The data may also be formatted in any computer-readableformat. By further way of example only, image data may be stored asbitmaps comprised of grids of pixels that are stored in accordance withformats that are compressed or uncompressed, lossless (e.g., BMP) orlossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well ascomputer instructions for drawing graphics. The data may comprise anyinformation sufficient to identify the relevant information, such asnumbers, descriptive text, proprietary codes, references to data storedin other areas of the same memory or different memories (including othernetwork locations) or information that is used by a function tocalculate the relevant data.

The processor 120 may be any conventional processor, such as processorsfrom Intel Corporation or Advanced Micro Devices. Alternatively, theprocessor may be a dedicated device such as an ASIC. Although FIG. 1functionally illustrates the processor, memory, and other elements ofcomputer 110 as being within the same block, it will be understood bythose of ordinary skill in the art that the processor and memory mayactually comprise multiple processors and memories that may or may notbe stored within the same physical housing. For example, memory may be ahard drive or other storage media located in a server farm of a datacenter. Accordingly, references to a processor or computer will beunderstood to include references to a collection of processors orcomputers or memories that may or may not operate in parallel.

Computer 110 may include all of the components normally used inconnection with a computer such as a central processing unit (CPU),memory (e.g., RAM and internal hard drives) storing data 126 andinstructions such as a web browser, an electronic display 134 (e.g., amonitor having a screen, a small LCD touch-screen or any otherelectrical device that is operable to display information), and userinput (e.g., a mouse, keyboard, touch-screen and/or microphone).

Computer 110 may also include a geographic position component 136 todetermine the geographic location of the device. For example, computer110 may include a GPS receiver to determine the device's latitude,longitude and/or altitude position. Other location systems such aslaser-based localization systems, inertial-aided GPS, WiFi or cellularsignal aided GPS, or camera-based localization may also be used.

Computer 110 may also include other features, such as an accelerometer,gyroscope or other acceleration device 140 to determine the direction inwhich the device is oriented. By way of example only, the accelerationdevice may determine its pitch, yaw or roll (or changes thereto)relative to the direction of gravity or a plane perpendicular thereto.In that regard, it will be understood that a computer's provision oflocation and orientation data as set forth herein may be providedautomatically to the user, other computers of the network, or both.

Computer 110 may also include an object detection component 142 todetect and identify the location and movement (e.g. relative speed) ofobjects such as other vehicles, obstacles in the roadway, trafficsignals, signs, etc. The detection system may include lasers, sonar,radar, cameras or any other such detection methods. For example, theobject detector may include an imaging device to identify the state of aparticular traffic signal as yellow or another color. In use, computer110 may use this information to instruct the braking system of thevehicle to apply the brakes and to provide information regarding suchobjects to the passenger of the vehicle, as described further below.

Data 126 may include various types of information used by computer 110.Detailed map information 136 may include maps identifying lane lines,intersections, speed limits, traffic signals, buildings, signs, or othersuch information. For example, computer 110 may access detailed mapinformation 136 in order to determine where the lane lines should belocated on a particular highway and adjust the speed or direction ofvehicle 101 accordingly. Computer 110 may also access display images130, such as roadways, intersections, and other objects in order toprovide a passenger of vehicle 101 with an understanding of what actionsvehicle 101 will take in the immediate future.

In one example, computer 110 may be an autonomous driving computingsystem capable of communicating with a vehicle's internal computer suchas computer 160. Computer 160 may be configured similarly to computer110, for example, including a processor 170, memory 172, instructions174, and data 176. Computer 110 may send and receive information fromthe various systems of vehicle 101, for example the breaking 180,acceleration 182, signaling 184, and navigation 186 systems in order tocontrol the movement, speed, etc. of vehicle 101. It will be understoodthat although various systems and computers 110 and 160 are shown withinvehicle 101, these elements may be external to vehicle 101 or physicallyseparated by large distances.

FIG. 2 depicts an exemplary design of the interior of an autonomousvehicle. Vehicle 101 may display information to a passenger in a numberof ways. For example, vehicle 101 may be equipped with an electronicdisplay 134 for the autonomous driving system and an indicator 144 whichidentifies whether the autonomous driving system has been engaged.Vehicle 101 may also identify the current speed of travel 138 bydisplaying the information in a location visible to the passenger oridentifying the speed audibly. Further, as shown in the figure, vehicle101 may also include a second display 188 for displaying additionalinformation such as that related to navigation functionality of thecomputer 110 (which can be used by the passenger when driving himself ina manual mode, or can be displayed to inform the user of a route usedfor autonomous driving in an autonomous mode). The display 188 can alsodisplay information related to an in-vehicle entertainment system.

Vehicle 101 may include one or more user input devices, such as device132, for inputting information into the autonomous driving computer 110.For example, a user may input a destination, (e.g. 123 Oak Street), intothe navigation system. The navigation system may generate a routebetween the present location of the vehicle and the destination. If theautonomous driving system is engaged, computer 110 may request orautomatically receive the route information from the navigation system.Once a route has been determined, the autonomous driving system maydrive the vehicle to the destination.

FIG. 3 is an exemplary screen shot of the display 134 of computer 110.The display may be used to identify to a passenger of the vehicle thecurrent speed, location, orientation, etc. of vehicle 101 with respectto a relevant stretch of roadway 105. For example, vehicle 101 isdepicted as a subject vehicle icon 112 on the screen. Any relevantobject may be used to identify the vehicle to the user. The route alongwhich the vehicle is traveling may be identified by a route line 115extending from the vehicle. In the example of FIG. 3, the vehicle ismoving forward, so route line 115 extends from the front of the vehicle.The display may also identify the speed indicator 138 of the device aswell as the approximate path of the vehicle during the next few seconds,shown as path 116. The path 116 can also be configured to represent ahead space in the direction of travel, as described further in U.S. Pat.No. 8,346,426, the entire disclosure of which is incorporated byreference herein.

As shown in FIG. 4, the display may also identify other relevantobjects. For example, where computer 110 has identified an object on theroadway (an “identified object”) as another vehicle, the computer maydisplay the vehicle as another icon, such as boxes 113, 113 a, on thedisplay in the approximate location of the roadway. The shape and sizeof the icon displayed may be selected by computer 110 based on the shapeand size of actual vehicle, for example a compact vehicle may appearsmaller than a tractor-trailer.

As vehicle 101 moves along the roadway, the location of objects detectedby the vehicle, and the features of the roadway may change. Thesechanges may be displayed in order to allow the user to understand thatvehicle 101 is continuously monitoring the state of the vehicles,roadway and other objects. For example, as shown in FIG. 4, vehicle box112 is driving along the roadway approaching a second vehicle identifiedas box 113 a. Some time later, vehicle 101 moves closer to the secondvehicle, and as shown in FIG. 5, the distance between vehicle box 112and box 113 a has decreased.

In addition to displaying representations (i.e. subject vehicle icon112) of the vehicle 101, and additional representations of othervehicles, such as vehicle boxes 113, the computer 110 can be configuredto provide warning information, via either display 134 or second display188, regarding vehicles, or other objects, that are identified as beingpotentially problematic for vehicle 101. Potentially problematic objectscan be those which are determined to be following too closely, vehiclespositioned in front of vehicle 101 that are rapidly decelerating (e.g.,braking abruptly), or other vehicles that are within a projected path ofvehicle 101 or are potentially entering into the path of vehicle 101 ina way such that they may potentially collide with vehicle 101.

Other objects, such as vehicles, bicycles, or pedestrians, can beidentified as potentially problematic simply based on their locationrelative to the vehicle 101. For example, a potentially problematicvehicle can be one that is driving in the blind spot of vehicle 101. Theblind spot of a vehicle is generally identified as a location alongsideof the vehicle that is between the line of sight of the rearview mirrorand the adjacent side-view mirror. In an example a warning relating to avehicle in the blind spot of vehicle 101 can be beneficial while thepassenger of the vehicle is driving to prevent collision between shouldthe driver wish to change lanes. In another example, a blind-spotwarning can be beneficial to the passenger while the car is in anautonomous driving mode to notify the passenger of, for example, thevehicle's delay in changing lanes according to an otherwisepredetermined autodriving path. Further, simply communicating suchnotifications to the passenger can give the passenger comfort in thefact that the computer is monitoring for such conditions, giving thepassenger confidence in the autonomous driving system 100.

As shown in FIG. 7, the computer 110 can be configured, such byappropriate programming or the like, to present a notification image 146on the display 134. In the example shown, notification image 146 ispresented as a segmented circle that surrounds subject vehicle icon 112.Subject vehicle icon 112 can take any form, and can be configured to beidentifiable by the drive as the subject vehicle 101. As shown in thefigures, the representation of subject vehicle icon 112 does not have toresemble a vehicle and can be a shape such as a triangle or an arrowhead(sometimes referred to as a “chevron”). Notification image 146 cansurround subject vehicle icon 112 and can include segments 148 a-148 d,that correspond to predetermined peripheral areas surrounding thevehicle such as in front of the vehicle 101 (segment 148 a), to the rearof the vehicle 101 (segment 148 c), the driver side (segment 148 b) andpassenger side (segment 148 d) of the vehicle 101.

The segments 148 of the notification image 146 can be present, whetheror not a specific warning is being communicated to the passenger bycomputer 110. In an example, when no warning is being communicated, thesegments 148 can have an appearance to notify the passenger that noproblematic objects are present. This can include presenting segments148 in a color that is accepted to indicate a positive or neutral systemstatus, such as green or blue, for example.

When a warning is to be communicated to the passenger of the vehicle101, the segment 148 corresponding to the position of the objectidentified as potentially problematic (the “identified object”) canchange to a warning color, such as yellow, orange, or red, for example.FIG. 7 shows segment 148 d with different shading from the remainingsegments so as to represent such a difference in color. In one example,different colors can be used depending on the urgency of the potentialproblem associated with the identified object. In this example, yellowcan be used to simply draw the passenger's attention to an identifiedobject that does not pose an immediate problem, but that could upon achange in condition. For example, if an identified vehicle is present inthe passenger side blind spot of vehicle 101, computer 110, can presentsegment 148 d in yellow. If the vehicle 101 (such as when being drivenby the passenger in a manual mode) begins to merge into the lane of theidentified vehicle, segment 148 d can be changed to orange, and if acollision is imminent, segment 148 d can be changed to red. System 100can be further configured to allow computer 110 to present an audiblewarning to the passenger, such as a horn or alarm, in the event of anelevated warning.

The manner in which the segments 148 a-148 d are associated with thepredetermined peripheral positions around the vehicle 101 can vary. Inone example, the divisions between the segments 148 can extend radiallyoutward from the center of subject vehicle icon 112 to divide the areasurrounding vehicle 101 into quadrants. In this example, the segment 148associated with such a quadrant can change in appearance when a warningis to be presented in connection with an identified object in thatquadrant. In another example, the areas associated with the segments 148can overlap, such that an identified object that is, for example,positioned in front of the vehicle 101 to the passenger side thereof,can be associated with both segments 148 a and 148 d such that if thatvehicle is an identified object for which a warning condition is to bepresented, both segments 148 a and 148 d can change in appearance (e.g.,color). In these or other possible configurations, multiple segments canchange in appearance simultaneously to indicate multiple warningsassociated with separate identified objects, such as a braking car tothe front of the vehicle 101 and a car in a blind spot of the vehicle101.

Computer 110 can be further configured to present a warning image inconnection with the change of appearance of a segment, such as segment148 d in the example above. For example, as shown in FIG. 8, warningimage 150 can be a simple symbol used to represent a warning, such as anexclamation point within a triangle shown. In another example, thewarning image 150 can be configured to identify a characteristic of theidentified object associated with the image. For example, and a shown inFIG. 9, if the computer 110 determines that the identified object is acar, an image of a car of the warning image 250, can be presented alongwith the indication provided by segments 248 d. In other examples,representations of trucks, pedestrians, animals, cyclists, or the likecan be presented to alert the passenger to identified objects of these(or other types), in such instances where computer 110 and the relatedobject sensors 142 (described above) are sensitive or accurate enough tomake such identifications.

As shown in FIGS. 10-13, notification image 146 can be presented in anumber of different ways. In the example of FIG. 10, notification image346 can be a plurality of unsegmented signals that can change colorwithin predetermined areas 348 to indicate a warning. Further theplurality of rings 152 can be concentrically arranged to indicate theproximity of the identified object with which the notification isassociated. That is, closer objects can be signaled to the passenger bya color change within an area 348 a of in inner ring 352 a, whilefarther objects can be signaled to the passenger by a color changewithin an area 348 c of outer ring 352 c.

In another example, FIG. 11 shows notification images presented as agrid over which subject vehicle icon 412 is presented. As shown, thegrid can have multiple layers, such as inner layer 452 a and outer layer452 b that are respectively closer to and farther from subject vehicleicon 412. The varying layers (which can include more than the two shownin FIG. 11) can be used to indicate the proximity of the identifiedobject to the vehicle 101, as discussed above in the example employingconcentric circles. That is, an identified object within a predetermineddistance of vehicle 101 (e.g. 10 feet) can be associated with a section448 of the grid within inner layer 452 a, and an identified objectoutside of such a predetermined distance can be associated with asection 448 within outer layer 452 b. The particular section 448 withinsuch a layer can be determined by associating the sections 448 withcorresponding peripheral areas of the vehicle 101 according to theprinciples described above with respect to FIG. 8.

In another example, FIG. 12 shows a notification image 546 that issimilar to that of FIG. 11, but with only a single layer of sections 548set out in a grid surrounding subject vehicle icon 512. The sections 548can be used to indicate the presence of a problematic identified objectwithin an area designated by the respective sections 548 in the samemanner as those discussed with respect to FIG. 11, but without thelevels of proximity indication.

FIG. 13 shows another example of a notification image 646 that is in theform of an unsegmented square that encircles subject vehicle icon 612.Various areas 648 of the image 646 can be altered (such as by a changein color, as described above) to indicate the presence of a potentiallyproblematic identified object within corresponding peripheral areas ofvehicle 101 in a manner similar to the circles of FIG. 10. Further, thesize of the areas 648, represented by the distance around image 646 thatthe areas 648 extend, can be presented in a way to indicate the size ofthe object (e.g., greater size for larger objects). In a variation,concentric squares could be presented in a manner similar to the circlesof FIG. 10. Additionally or alternatively, the square could besegmented, as in the example of FIGS. 6-8. This or any of the othervarious notification images in FIGS. 9-13 can be presented on display134 along with any of the warning images 150 shown in FIGS. 7 and 8.

Any of the various notification images shown in FIGS. 6-13 can bepresented to the user on either display 134 or second display 188. Inone example, the subject vehicle icon 112, the notification image 146,and any warning images 150 (if needed at any point) can be presented onprimary display 134. Simultaneously with the images being presented onprimary display 134 (or in the alternative thereto), subject vehicleicon 112, notification image 146 and any needed warning images 150 canbe presented on secondary display 188. In the example shown in FIG. 14,these images can be incorporated into the presentations of the roadwayimages shown in FIGS. 3-5. That is, the subject vehicle icon 112 can beshown on the road image 105 in the location of vehicle 101 as determinedby the system 100 using the global positioning components 136 thereof.Subject vehicle icon 112 can be presented over route line 115 and overpath 116. Notification image 146 can be presented along with subjectvehicle icon 112 surrounding it as if projected on the road image 105.As discussed above, other objects, such as vehicles, as identified bythe object sensors of system 101 can also be represented on road 105 inthe positions thereof relative to the vehicle 101 as boxes 113 on road105.

As shown in FIG. 14, when none of the other vehicles shown on the roadimage by boxes 113 have been identified by computer 110 as beingpotentially problematic to vehicle 101, notification image 146 can beshown in a non-warning state (such as all-green as discussed above).Similarly, if an identified object, such as object 113 a in FIG. 15 isin front of vehicle 101, as indicated by subject vehicle image 112, atan appropriate distance, notification image 146 can be shown in asimilar, non-warning state. If an identified object is determined by thesystem 101 to be potentially problematic, one or more segments 148 ofnotification image 146 can change in appearance to indicate such awarning state in an area associated with that segment, as discussedabove. Further, the particular identified object that triggers such anotification image can be indicated by computer 110 on display 134. Asshown in FIG. 16, if the vehicle indicated by box 113 a, which isdriving in front of vehicle 101, slows or brakes abruptly such thatvehicle 101 should slow down accordingly or take action to avoid acollision, segment 148 a can change in color, as discussed above, toindicate such a warning. Further, a warning image 150 can be presentedin front of segment 148 a, as shown in FIG. 16, and box 113 a can bepresented in a color to indicate that the warning image andcorresponding notification are associated with a vehicle in front ofvehicle 101. In an example, this can include a color change (asindicated by the shading tone of box 113 a in FIG. 16) that can, forexample, be to the same color as segment 148 a and/or warning image 150.

In one example, the above-described warning can be used to alert thedriver of vehicle 101 to a condition to help the driver avoid anaccident or the like. In another example, when the system 101 is engagedin autonomous driving, such notifications or warning images can helpcommunicate to a passenger the reasons for system 101 taking variousactions. In the example shown in FIG. 15, the warning image, segmentvisual indication, and vehicle identification can be implemented asvehicle 101 approaches another vehicle, represented by box 113 a, frombehind. In such an instance, computer 110 may maintain an appropriatefollowing distance and display a variation of the above-describednotifications to the passenger. For example, in FIG. 15 subject vehicleicon 112 is shown as moving along roadway 105. As subject vehicle icon112 approaches box 113 a, box 113 a may be changed, for example toyellow (or flash yellow, etc.). Simultaneously, segment 148 a can changeto yellow (or another appropriate color, for example). Such anindication can be used to notify the passenger as to the reason forvehicle 101 slowing down.

In another example, should the vehicle represented by box 113 a brake orotherwise slow abruptly, vehicle 101, if engaged in autonomous driving,can automatically to do the same. In connection with this action, anotification can be made (and can in an example be presented in anurgent manner such as in red or the like) so that the passenger is madeaware of the reason for the abrupt braking by vehicle 101. As shown inFIG. 16, a corresponding warning 150 image can be presented directly infront of segment 148 a. In the alternative, the system can refrain frompresenting warning images during autonomous driving, instead onlychanging the color of the relevant segment 148 and the box 113associated with the identified object. In another example, a warningimage 750 can be shown adjacent box 713 a, as shown in FIG. 17.

In another example, a similar procedure to that shown in FIGS. 16 and 17can be used to alert the passenger of vehicle 101 to the presence ofanother vehicle approaching the rear of vehicle 101 or otherwise drivingat an unsafe distance with respect to vehicle 101. Such a distance canbe determined as an absolute distance (e.g., 10 feet) or can bedetermined by the speed of vehicle 101 or the other vehicle such as bydetermining a time based following distance. Such a following distancecan be determined by the time it would take a following object to reachan instantaneous location of a followed object, given the followingobject's speed. In an example, a warning indication can be given whenanother vehicle at a two second following distance or less relative tovehicle 101. This type of warning indication can alert the driver to thereason for a corresponding action taken by system 100, such as changinglanes or speeding up (if conditions and/or applicable speed limitspermit). Such notifications can be given simultaneously with or justbefore actions are taken by the system 100, for example. A similarwarning can be given to alert the passenger or driver to a stationaryobject to the rear of the vehicle 101 when it is in reverse.

Computer 110 may also display warnings or indications as discussedabove, using notification image 146 and optionally warning images 150 inconnection with identified objects in order to inform the passenger thatthe computer will not take a particular action because of the presenceof that identified object. As shown in FIG. 18, vehicle 101 isapproaching an intersection. The shape of path 116 indicates thatcomputer 110 will make vehicle 101 perform a left turn. In the example,computer 110 has identified a vehicle, displayed as box 113 b which willpass through path 116. Computer 110 may change the color of box 113 band of segment 148 b and may further present a warning image 150adjacent box 113 b (or segment 148 b) indicating that computer 110 willnot move through the intersection until the identified object associatedwith box 113 b is no longer within the path of the vehicle 101. The sametype of warning can also be displayed, for example, in connection with apedestrian or the like crossing the intersection in the path 116 ofvehicle 101.

Computer 110 may also use the display to indicate to the passenger thatthe vehicle 101 will be changing lanes, but is waiting to do so becauseof an identified object that makes changing lanes unsafe. As shown inFIG. 19, vehicle 101 is moving along a route 115 indicated by path 116behind a second vehicle, displayed as box 113 a. The presentation ofsegment 148 a in green, for example, may indicate that vehicle 101 ismaintaining an appropriate following distance along path 116 behind thesecond vehicle. If computer 110 determines a change of lanes isappropriate as shown in FIG. 20, the computer may display path 116 so asto change from within the current lane to the lane to which computer 110intents to move vehicle 101. This can be done, for example, if thevehicle's route requires the vehicle to take an exit, make a turn, etc.If a third vehicle is present within the lane into which computer 110would like to move, such as the identified object indicated by box 113d, computer will refrain from changing lanes. In some instances,computer may simply wait for the identified object to move from out ofan area that makes changing lanes unsafe or may cause vehicle 101 tospeed up or slow down (conditions permitting) to position vehicle 101relative to the identified object so as to make changing lanesappropriate. In either instance, while computer 110 is taking actionthat includes refraining from changing lanes because of the identifiedobject indicated by box 113 d, computer 110 can display the notificationimage shown in FIG. 19. As discussed above, this can include changingsegment 148 d to yellow and can optionally further include changing box113 d to yellow (for example) as well and/or presenting a warning image150 adjacent segment 148 d or box 113 d. As vehicle 101, for example,begins to slow down to position itself to appropriately change lanes,computer 110 may maintain the notification state, until changing lanesis appropriate, at which point, the notification image 146 can return toa non-warning, or neutral (e.g., all green) image and any warning imagecan be removed simultaneously with or just before computer 110 causesvehicle 101 to change lanes.

Additional warning information, such as an escalated warning state asdescribed above can further be presented in the above scenario, forexample, if the identified object indicated by box 113 d itself beginsto unsafely change lanes into the lane in which vehicle 101 istraveling. This can be done in connection with computer 110 causingvehicle 101 to take action to avoid a collision with the identifiedobject. Other similar notifications and warnings can also be presentedby computer 110 according to other criteria, such as a vehicle movingunsafely into the path of the subject vehicle 100 (either determined bythe projected route or inferred by the system 101 during non-autonomousdriving) or otherwise being detected within the path. Further, thenotification scheme described herein can also alert the driver to theending of the lane of travel, for example, while driving on a highway.It can also alert the driver to unsafe or illegal actions to be taken(inadvertently or by error) by the driver during non-autonomous driving.In one example vehicle 101 could present such a warning to the driverwhen the driver begins to make an illegal turn or the like.

As these and other variations and combinations of the features discussedabove can be utilized without departing from the invention as defined bythe claims, the foregoing description of exemplary embodiments should betaken by way of illustration rather than by way of limitation of theinvention as defined by the claims. It will also be understood that theprovision of examples of the invention (as well as clauses phrased as“such as,” “e.g.”, “including” and the like) should not be interpretedas limiting the invention to the specific examples; rather, the examplesare intended to illustrate only some of many possible aspects.

The sample values, icons, types and configurations of data described andshown in the figures are for the purposes of illustration only. In thatregard, systems and methods in accordance with aspects of the inventionmay include different physical attributes, data values, data types andconfigurations, and may be provided and received at different times andby different entities (e.g., some values may be pre-suggested orprovided from different sources).

The invention claimed is:
 1. A method comprising: identifying, by one or more processors, a notification image including a representation of a vehicle and a plurality of segments representing physical areas in an environment around the vehicle; detecting, by the one or more processors, an object at a particular location in the environment using a sensor system; identifying, by the one or more processors, a segment of the plurality based on the particular location; generating, by the one or more processors, a visual representation of the environment of the vehicle, the visual representation including a path of the vehicle and a representation of the vehicle; and displaying, by the one or more processors, the visual representation, the representation of the vehicle, the path and the identified segment such that the representation of the vehicle appears to be overlaid on the path, and the identified segment is depicted as a highlighted area adjacent to the representation of the vehicle in order to identify a location of the object relative to the vehicle.
 2. The method of claim 1, further comprising displaying all of the segments of the plurality with the visual representation, the representation of the vehicle, and the path.
 3. The method of claim 2, wherein the identified segment is displayed in a different color from other displayed segments of the plurality.
 4. The method of claim 1, wherein each of the segments of the plurality extend radially outward from the vehicle.
 5. The method of claim 1, wherein the identified segment is displayed as an area that extends radially outward from the vehicle.
 6. The method of claim 1, further comprising: detecting a second object at a second particular location in the environment; and identifying a second segment of the plurality based on the second particular location, and displaying the second identified segment is displayed the visual representation, the representation of the vehicle, the path and the identified segment, the second identified segment being displayed as a highlighted area adjacent to the representation of the vehicle in order to identify a location of the second object relative to the representation of the vehicle.
 7. The method of claim 1, further comprising: detecting a second object, the second object corresponding to a second vehicle; and displaying a representation of the second object in front of the visual representation relative to the path.
 8. The method of claim 1, wherein an area of the identified segment overlaps with an area of another segment of the plurality of segments.
 9. The method of claim 1, wherein displaying the identified segment includes displaying an image of an object representative of a type of the object.
 10. A system comprising one or more processors configured to: identify a notification image including a representation of a vehicle and a plurality of segments representing physical areas in an environment around the vehicle; detect an object at a particular location in the environment using a sensor system; identify a segment of the plurality based on the particular location; generate a visual representation of the environment of the vehicle, the visual representation including a path of the vehicle and a representation of the vehicle; and display the visual representation, the representation of the vehicle, the path and the identified segment such that the representation of the vehicle appears to be overlaid on the path, and the identified segment is depicted as a highlighted area adjacent to the representation of the vehicle in order to identify a location of the object relative to the vehicle.
 11. The system of claim 10, further comprising the vehicle.
 12. The system of claim 10, wherein the one or more processors are further configured to display all of the segments of the plurality with the visual representation, the representation of the vehicle, and the path.
 13. The system of claim 12, wherein the one or more processors are further configured to display the identified segment in a different color from other displayed segments of the plurality.
 14. The system of claim 10, wherein each of the segments of the plurality extend radially outward from the vehicle.
 15. The system of claim 10, wherein the one or more processors are further configured to display the identified segment as an area that extends radially outward from the vehicle.
 16. The system of claim 10, wherein the one or more processors are further configured to: detect a second object at a second particular location in the environment; and identify a second segment of the plurality based on the second particular location, and displaying the second identified segment is displayed the visual representation, the representation of the vehicle, the path and the identified segment, the second identified segment being displayed as a highlighted area adjacent to the representation of the vehicle in order to identify a location of the second object relative to the representation of the vehicle.
 17. The system of claim 10, wherein the one or more processors are further configured to: detect a second object, the second object corresponding to a second vehicle; and display a representation of the second object in front of the visual representation relative to the path.
 18. The system of claim 10, wherein an area of the identified segment overlaps with an area of another segment of the plurality of segments.
 19. The system of claim 10, wherein displaying the identified segment includes displaying an image of an object representative of a type of the object.
 20. A non-transitory, machine readable medium on which instructions are stored, the instructions when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: identifying a notification image including a representation of a vehicle and a plurality of segments representing physical areas in an environment around the vehicle; detecting an object at a particular location in the environment using a sensor system; identifying a segment of the plurality based on the particular location; generating a visual representation of the environment of the vehicle, the visual representation including a path of the vehicle and a representation of the vehicle; and displaying the visual representation, the representation of the vehicle, the path and the identified segment such that the representation of the vehicle appears to be overlaid on the path, and the identified segment is depicted as a highlighted area adjacent to the representation of the vehicle in order to identify a location of the object relative to the vehicle. 